Image processing apparatus and image processing method

ABSTRACT

An image processing apparatus including: an examination image obtaining unit obtaining an examination image of an examination subject; a shift-invariant feature quantity calculating unit calculating, for each pixel, a shift-invariant feature quantity represented by predetermined base vectors, from the examination image obtained by the examination image obtaining unit; a selecting unit selecting, on the examination image, a pixel having a matching degree lower than or equal to a predetermined threshold, between (i) a relative positional relationship of classes in normal images each of which does not include a lesion site and (ii) a relative positional relationship of the classes to which shift-invariant feature quantities respectively belong in the examination image, the classes being obtained by clustering the shift-invariant feature quantities: calculated from pixels included in the normal images; and represented by the predetermined base vectors; and an output unit outputting a result of the selection performed by the selecting unit.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation application of PCT International Application No.PCT/JP2013/002988 filed on May 9, 2013, designating the United States ofAmerica, which is based on and claims priority of Japanese PatentApplication No. 2012-119079 filed on May 24, 2012. The entiredisclosures of the above-identified applications, including thespecifications, drawings and claims are incorporated herein by referencein their entirety.

FIELD

Apparatuses and methods consistent with one or more exemplaryembodiments of the present disclosure relate to an image processingapparatus and an image processing method for processing a medical image.

BACKGROUND

In recent years, there has been increase in the importance ofcomputer-aided diagnosis (CAD) for supporting diagnosis by a doctor withthe image analysis techniques using a computer. The most generalfunction of the CAD is detection of a lesion site in an image.

When detecting a lesion using a computer, an examination image and anormal image are compared to find a difference. Calculation of adifference between two data items is a fundamental function in computerprocessing, and the computer processing excels at diagnostic imagingwhich detects a lesion site by comparing an examination image and anormal image which is a medical image of a normal structure.

However, position adjustment of the normal image and the examinationimage needs to be carried out for performing the difference calculation.Patent Literature (PTL) 1 discloses a method with which a landmark isset to each of the normal image and the examination image, positionadjustment is carried out for the normal image and the examination imageso as to match the landmarks, and a lesion site is detected based on thedifference between the normal image and the examination image, therebysupporting diagnostic imaging.

CITATION LIST Patent Literature

-   [PLT 1] Japanese Unexamined Patent Application Publication No.    2004-41694

SUMMARY

However, in PLT 1, there is no mentioning about a method of setting thelandmark. Position adjustment can be done in various aspects accordingto the difference between setting methods of the landmark, variation ina setting position of the landmark, and so on, even with the samecombination of the normal image and the examination image. For thatreason, the lesion site is detected or not detected depending on theaspect of the position adjustment, leading to a problematic situation inwhich detecting of a lesion cannot be accurately performed.

One or more exemplary embodiments of the present disclosure provide animage processing apparatus capable of performing accurate lesiondetection.

In one general aspect, the techniques disclosed here feature an imageprocessing apparatus including: an examination image obtaining unitwhich obtains an examination image which is an image of an examinationsubject; a shift-invariant feature quantity calculating unit whichcalculates, for each pixel, a shift-invariant feature quantity which isrepresented by predetermined base vectors, from the examination imageobtained by the examination image obtaining unit; a selecting unit whichselects, on the examination image, a pixel having a matching degreelower than or equal to a predetermined threshold, between (i) a relativepositional relationship of classes in a plurality of normal images eachof which does not include a lesion site and (ii) a relative positionalrelationship of the classes to which a plurality of shift-invariantfeature quantities respectively belong in the examination image, theclasses being obtained by clustering the plurality of shift-invariantfeature quantities which are: calculated from a plurality of pixelsincluded in the normal images; and represented by the predetermined basevectors; and an output unit which outputs a result of the selectionperformed by the selecting unit.

It is to be noted that these generic and specific aspects may beimplemented using a system, a method, an integrated circuit, a computerprogram, or a non-transitory computer-readable recording medium such asa compact disc read only memory (CD-ROM), and may also be implemented byany combination of systems, apparatuses, methods, integrated circuits,computer programs, and recording media.

According to various exemplary embodiments of the present disclosure, itis possible to perform accurate lesion detection.

BRIEF DESCRIPTION OF DRAWINGS

These and other advantages and features of the present disclosure willbecome apparent from the following description thereof taken inconjunction with the accompanying Drawings that illustrate general andspecific exemplary embodiments of the present disclosure. In theDrawings:

FIG. 1 is a diagram for describing a landmark setting according to PLT1.

FIG. 2 is a block diagram illustrating a configuration of an imageprocessing apparatus according to one exemplary embodiment.

FIG. 3 is a block diagram which illustrates a configuration of an imageprocessing apparatus for generating data to be used by the imageprocessing apparatus illustrated in FIG. 2.

FIG. 4 is a flowchart illustrating a procedure for generating a normalstructure database.

FIG. 5 is a diagram illustrating an example of calculatingshift-invariant image feature quantity according to wavelettransformation.

FIG. 6A is a diagram illustrating an example of calculating a waveletcoefficient employing Haar's mother wavelet.

FIG. 6B is a diagram illustrating an example of calculating a waveletcoefficient employing Haar's mother wavelet.

FIG. 7 is a diagram illustrating an example of a normal image featurequantity vector.

FIG. 8 is a diagram illustrating an example of transforming the normalimage feature quantity vector into a class number.

FIG. 9 is a diagram illustrating an example of a local feature normalstructure database.

FIG. 10 is a diagram illustrating an example of the case where a lesionsite cannot be determined correctly using only the local featurequantity.

FIG. 11 is a diagram illustrating an example of the case where arelative positional relationship between the local feature quantities isemployed as the structure relationship.

FIG. 12 is a diagram illustrating an example of a relative positionalrelationship normal structure database (relationship with the class of aneighboring right pixel).

FIG. 13 is a diagram illustrating an example of a relative positionalrelationship normal structure database (relationship with a neighboringright class).

FIG. 14 is a flowchart illustrating a procedure of lesion detectingprocessing.

FIG. 15 is a diagram illustrating an example of a procedure ofdetermining a threshold to be used in lesion determination.

DETAILED DESCRIPTION Underlying Knowledge Forming Basis of the PresentDisclosure

The inventors have found the following problems related to the methoddisclosed by PLT 1 described in the “Background” section.

As stated above, position adjustment of the normal image and theexamination image needs to be carried out for performing the differencecalculation. In generating a normal image, a medical image capturedpreviously is used in general. When a lesion site is not found by anexamination of the examination image by an image interpreter, the imageis determined as a normal image. More specifically, when there is nolesion site found in an examination image of a patient which waspreviously captured, it is possible to use the examination image as thenormal image. However, it is often the case that the position of anorgan or the like differs between the normal image and the examinationimage of even the same patient, due to a variety of factors such as thedifference in image capturing conditions, a change in the body shape ofa patient, and so on. In addition, when an examination image is capturedfor the first time, a normal image of the same patient is not presentwhich is to be a subject for comparison. In such a case, a normal imageof a different patient will be used as a subject for comparison, andthis requires position adjustment of the normal image and theexamination image due to the difference of the body shape of thedifferent patient.

To be more specific, the position adjustment is implemented by geometrictransformation such as rotation, parallel translation, enlargement,reduction, and so on. For example, a plurality of corresponding pointsare set to the normal image and the examination image, and affinetransformation and enlargement or reduction are performed on one of theimages such that the corresponding points of the images match.

By the way, a normal image is generated from an examination image whichwas previously captured and confirmed that there is no lesion. Thereason for that is because, first, there is no medical image which canbe a subject to be compared with an examination image of a patient whoseimage is captured for the first time, as described above. Secondly,medical knowledge tends to be established by piling up previous cases.The medical utility value is likely to be higher when generated fromprevious cases, also for a normal image which does not include a lesion.The medical knowledge is constantly advancing, and interpretation of theprevious cases is modified in some cases. The medical knowledgeregistered on an IT system constantly requires update, and the normalimage is no exception.

In view of the above, it is desirable to collect normal images of aplurality of patients and to generate a normal image with a highversatility to represent the collected normal images cyclopaedically. Asa specific method to accomplish this, for example, PLT 1 discloses thata normal image is represented by a linear combination of an averageshape and an eigen shape. In sum, the shape vector representing a normalstructure is expressed by Expression 1 indicated below.[Math. 1]x=x _(ave) +Ps·bs  Expression 1

Here, X_(ave) represents an average shape vector, Ps represents an eigenshape vector, and bs represents a set of shape coefficients.

The average shape vector X_(ave) and the eigen shape vector Ps arenecessary for the calculation of Expression 1, and a landmark M asillustrated in FIG. 1 is set on an image for vectorizing imageinformation. In the diagram, the black dots are the landmarks M. Thecoordinates x and y of the landmarks M are elements of a vector, and theimage information is vectorized. As shown in the examination images P1,p2, and P3, landmarks are set and a shape vector is defined individuallyfor a plurality of normal images, and an average shape vector and aneigen shape vector are calculated from them. It is to be noted that theexamination images are also expressed by Expression 1 in the samemanner.

Position adjustment is carried out on the normal image and theexamination image using the vectors described above, and a lesion siteis detected from a difference between the normal image and theexamination image, thereby supporting the diagnostic imaging.

However, the operation of setting a landmark is cumbersome according tothe method described in PLT 1, leading to decreased efficiency ofdiagnosis practice. In addition, since the coordinate x and thecoordinate y of the landmark changes and the vector element changes bychanging the method of setting a landmark, the average shape vector andthe eigen shape vector, as a result, differ from the average shapevector and the eigen shape vector before the change in the method ofsetting. PLT 1 fails to disclose the method of setting a landmark, andtherefore various normal images (shape vectors representing the normalstructure) are generated due to the difference in the method of settinga landmark and the variation of setting position of a landmark even whenthe same technique is employed. Medical knowledge is established byaccumulating previous cases, and thus, in terms of reusability, it isproblematic that a single case is defined in many ways according to themethod of setting a landmark. As described above, it poses a problem ofdecrease in determination accuracy of a lesion site when a single caseis defined in many ways. In other words, the same examination image isdetermined as a lesion image or as a normal image depending on thedefinition.

The present disclosure enables detection of a lesion without requiringthe setting of a landmark for position adjustment, by describing astructure of a normal image using the shift-invariant feature quantity.Furthermore, by employing, as knowledge, a relative positionalrelationship between local image feature quantities, it is possible todetect a lesion more accurately, taking into consideration the structureof human body that cannot be represented only by a local feature.

In one general aspect, the techniques disclosed here feature an imageprocessing apparatus including: an examination image obtaining unitwhich obtains an examination image which is an image of an examinationsubject; a shift-invariant feature quantity calculating unit whichcalculates, for each pixel, a shift-invariant feature quantity which isrepresented by predetermined base vectors, from the examination imageobtained by the examination image obtaining unit; a selecting unit whichselects, on the examination image, a pixel having a matching degreelower than or equal to a predetermined threshold, between (i) a relativepositional relationship of classes in a plurality of normal images eachof which does not include a lesion site and (ii) a relative positionalrelationship of the classes to which a plurality of shift-invariantfeature quantities respectively belong in the examination image, theclasses being obtained by clustering the plurality of shift-invariantfeature quantities which are: calculated from a plurality of pixelsincluded in the normal images; and represented by the predetermined basevectors; and an output unit which outputs a result of the selectionperformed by the selecting unit.

According to this configuration, the shift-invariant feature quantitycalculated from a normal image and the shift-invariant feature quantitycalculated from an examination image are represented by the same basevectors. This eliminates the necessity of setting a landmark forposition adjustment between the normal image and the examination image.In addition, a pixel is selected using the matching degree between (i)the relative positional relationship of classes of the shift-invariantfeature quantities in an examination image and (ii) the relativepositional relationship of classes obtained from the result ofclustering the shift-invariant feature quantities in a normal image.Thus, by employing, as knowledge, a relative positional relationshipbetween the local image feature quantities, it is possible to detect alesion more accurately, taking into consideration the structure of humanbody that cannot be represented only by a local feature.

For example, the shift-invariant feature quantity calculating unit mayinclude: an image feature quantity calculating unit which calculates,for each pixel, an examination image feature quantity vector which has aplurality of the shift-invariant feature quantities as elements of avector, from the examination image obtained by the examination imageobtaining unit; and a base representing unit which calculates, for eachpixel of the examination image, (i) coefficients used to represent theexamination image feature quantity vector in a linear combination ofnormal image base vectors which are: base vectors of a plurality ofnormal image feature quantity vectors each of which is calculated fromthe pixels included in the normal images; and base vectors of the normalimage feature quantity vectors each having the shift-invariant featurequantities as the elements of a vector, and (ii) an examination imagebase coefficient vector having the calculated coefficients as theelements of the vector, the selecting unit may include: a nearestneighbor vector obtaining unit which (i) obtains, for the each pixel ofthe examination image, a local feature normal structure vector which ismost similar to an examination image base coefficient vector from alocal feature normal structure database, and (ii) obtains: a classnumber of a class to which the obtained local feature normal structurevector belongs; and a distance between the examination image basecoefficient vector and the obtained local feature normal structurevector, the local feature normal structure database storing, for each ofclasses obtained by clustering a plurality of normal image basecoefficient vectors, the local feature normal structure vector which isa center vector representing at least one normal image base coefficientvector that belongs to the class, together with a class number of theclass, the plurality of the normal image base coefficient vectors beingobtained from the normal image feature quantity vectors and each having,as elements, coefficients used to represent the normal image featurequantity vector in the linear combination of the normal image basevectors; a relative positional relationship obtaining unit which obtainsa structural matching degree of the each pixel of the examination image,using the class number obtained by the nearest neighbor vector obtainingunit, from a relative positional relationship normal structure databasewhich stores, as the structural matching degree, a relative positionalrelationship of the classes in the normal images, the relativepositional relationship being obtained from class numbers resulting fromclustering performed on the normal image feature quantity vectors; and apixel selecting unit which selects, on the examination image, a pixel ofwhich (i) the distance obtained by the nearest neighbor vector obtainingunit is greater than or equal to a first threshold or (ii) thestructural matching degree obtained by the relative positionalrelationship obtaining unit is smaller than or equal to a secondthreshold, and the output unit which outputs a result of the selectionperformed by the selecting unit.

According to this configuration, the examination image and the normalimage are compared using the base coefficient vectors of theshift-invariant feature quantities. For that reason, it is possible todetermine the presence or absence of a lesion without performing theposition adjustment between the normal image and the examination image.Since the position adjustment processing is not required, there is nodecrease in the accuracy of determining a lesion site caused by thedifference in a setting method of a landmark or variation in a settingposition, and thus it is possible to provide an image processingapparatus with high accuracy in determining a lesion site. Furthermore,a pixel, on an examination image, of which the structural matchingdegree indicating the relative positional relationship of classes in thenormal image is lower than or equal to the second threshold. For thatreason, it is possible to select a pixel, on the examination image,which has a structure different from a structure of the normal image.Thus, it is possible to detect a lesion more accurately, taking intoconsideration the structure of human body that cannot be representedonly by a local feature.

To be specific, the pixel selecting unit may include a lesiondetermining unit which determines, on the examination image, a pixel ofwhich (i) the distance obtained by the nearest neighbor vector obtainingunit is greater than or equal to a first threshold or (ii) thestructural matching degree obtained by the relative positionalrelationship obtaining unit is smaller than or equal to a secondthreshold, as a pixel of a lesion site, and

the output unit may output a result of the determination performed bythe lesion determining unit.

In addition, the relative positional relationship normal structuredatabase may store an appearance probability of classes of pixelslocated to be in a predetermined positional relationship, as thestructural matching degree, and the relative positional relationshipobtaining unit may (i) identify classes of the pixels located to be inthe predetermined positional relationship in the examination image,using the class number obtained by the nearest neighbor vector obtainingunit, and (ii) obtain the appearance probability of the classes of thepixels located to be in the predetermined positional relationship fromthe relative positional relationship normal structure database, toobtain the structural matching degree of each of the pixels included inthe examination image.

According to this configuration, by employing, as the matching degree,the appearance probability of classes of pixels located to be in apredetermined positional relationship in the normal image, it ispossible to detect, as a lesion site, a combination of classes which isnot often appear in the normal image when the combination appears in theexamination image.

In addition, the relative positional relationship normal structuredatabase may store, as a structural matching degree, an appearanceprobability of classes of pixels aligned in a predetermined directionand having different class numbers, and the relative positionalrelationship obtaining unit may obtain, from the relative positionalrelationship normal structure database, the appearance probability ofthe classes of pixels aligned in the predetermined direction and havingthe different class numbers in the examination image, using the classnumber obtained by the nearest neighbor vector obtaining unit, to obtaina structural matching degree of each of the pixels included in theexamination image.

The probability that the same class appears in adjacent pixels is highin some cases. However, according to this configuration, it is possibleto perform lesion detection, using the appearance probability of a classof pixels which are present at positions that are mutually distant tosome extent and have different class numbers, as the matching degree.For that reason, the structure of the examination image is more easilycaptured, and thus it is possible to increase the accuracy in the lesiondetection.

In another general aspect, the techniques disclosed here feature animage processing apparatus including: a normal image obtaining unitwhich obtains a plurality of normal images which are images including nolesion; an image feature quantity calculating unit which calculates, foreach pixel, a normal image feature quantity vector which has a pluralityof shift-invariant feature quantities as elements of a vector, from eachof the normal images obtained by the normal image obtaining unit; aprincipal component analysis unit which performs principal componentanalysis on the normal image feature quantity vectors calculated, by theimage feature quantity calculating unit, from pixels included in thenormal images to obtain (i) normal image base vectors which are basevectors of the normal image feature quantity vectors and (ii) aplurality of normal image base coefficient vectors resulting from baseconversion performed on the normal image feature quantity vectors usingthe normal image base vectors; a base vector output unit which writes,on a base vector storage, the normal image base vector obtained by theprincipal component analysis unit; a clustering unit which performsclustering on the normal image base coefficient vectors to obtain, foreach class, a center vector which represents at least one of the normalimage base coefficient vectors which belong to the class; a centervector output unit which writes, on a local feature normal structuredatabase, the center vector obtained by the clustering unit, togetherwith a class number of a class represented by the center vector; arelative positional relationship calculating unit which calculates, as astructural matching degree, a relative positional relationship ofclasses in the normal images, the relative positional relationship beingobtained from the class number; and a relative positional relationshipoutput unit which writes, as the structural matching degree, a relativepositional relationship calculated by the relative positionalrelationship calculating unit, between classes in the normal images, ona relative positional relationship normal structure database.

According to this configuration, it is possible to generate, asknowledge, a structural matching degree indicating a relative positionalrelationship of classes in a normal image. The image processingapparatus described above uses such knowledge, thereby making itpossible to detect a lesion accurately.

For example, the shift-invariant feature quantity may include a waveletcoefficient, a higher order local autocorrelation (HLAC) featurequantity, a scale-invariant feature transform (SIFT) feature quantity,or a histogram of oriented gradients (HOG) feature quantity.

In addition, the image may be one of a radiological image, an ultrasoundimage, and a pathological specimen image.

It is to be noted that these generic and specific aspects may beimplemented using a system, a method, an integrated circuit, a computerprogram, or a non-transitory computer-readable recording medium such asa compact disc read only memory (CD-ROM), and may also be implemented byany combination of systems, apparatuses, methods, integrated circuits,computer programs, and recording media.

Hereinafter, certain exemplary embodiments are described in greaterdetail with reference to the accompanying Drawings.

Each of the exemplary embodiments described below shows a general orspecific example. The numerical values, structural elements, thearrangement and connection of the structural elements, steps, theprocessing order of the steps etc. shown in the following exemplaryembodiments are mere examples, and therefore do not limit the scope ofthe appended Claims and their equivalents. Therefore, among thestructural elements in the following exemplary embodiments, structuralelements not recited in any one of the independent claims are describedas arbitrary structural elements.

FIG. 2 is a block diagram which illustrates a configuration of an imageprocessing apparatus according to the exemplary embodiment.

According to the exemplary embodiment, first as a preparation, anexamination image captured by a medical device is stored in anexamination image storage 300. An image processing apparatus 310 obtainsan examination image from the examination image storage 300 to detect alesion site from the obtained examination image, and outputs a result ofthe detection.

The image processing apparatus 310 includes: a base vector storage 130;a local feature normal structure database 150; a relative positionalrelationship normal structure database 170; a lesion determinationthreshold database 190; an examination image obtaining unit 100; ashift-invariant feature quantity calculating unit 105; a selecting unit106; and an output unit 200.

The base vector storage 130 stores a normal image base vectors (a normalimage base vector matrix, a normal image average vector). The normalimage base vectors are base vectors of a plurality of normal imagefeature quantity vectors calculated from a plurality of pixels includedin a plurality of normal images, and each of the normal image featurequantity vectors has, as elements, a plurality of shift-invariantfeature quantities

The local feature normal structure database 150 stores a plurality oflocal feature normal structure vectors calculated in advance from thenormal images. The local feature normal structure vector is a centervector which represents one or more normal image base coefficientvectors which belong to the respective classes obtained by clustering aplurality of normal image base coefficient vectors that will bedescribed later. Here, the normal image base coefficient vector has, aselements, coefficients used to represent the normal image featurequantity vector in a linear combination of the normal image basevectors.

The relative positional relationship normal structure database 170stores a structural matching degree between local feature quantitieswhich are local shift-invariance feature quantities calculated inadvance from a plurality of normal images. More specifically, therelative positional relationship normal structure database 170 stores,as the structural matching degree, a relative positional relationship ofclasses in a plurality of normal images obtained from class numbersresulting from the clustering performed on the normal image featurequantity vectors.

The lesion determination threshold database 190 stores a first thresholdand a second threshold as thresholds for determining a lesion.

It is to be noted that, the base vector storage 130, the local featurenormal structure database 150, the relative positional relationshipnormal structure database 170, and the lesion determination thresholddatabase 190 may be included in the image processing apparatus 310, ormay be disposed outside the image processing apparatus 310 and connectedto the image processing apparatus 310 via wired or wirelesscommunication. The local feature normal structure database 150, therelative positional relationship normal structure database 170, and thelesion determination threshold database 190 are storages each configuredby hardware or the like, and in the description below, data stored inthe storages also referred to as database.

The examination image obtaining unit 100 obtains an examination imagewhich is a medical image of an examination subject stored in theexamination image storage 300.

The shift-invariant feature quantity calculating unit 105 calculates,for each pixel, a shift-invariant feature quantity which is representedby a predetermined base vector, from the examination image obtained bythe examination image obtaining unit 100. Details of the shift-invariantfeature quantity calculating unit 105 will be described later.

The selecting unit 106 selects, on the examination image, a pixel havinga matching degree lower than or equal to a predetermined threshold,between (i) a relative positional relationship of classes and (ii) arelative positional relationship of the classes to which a plurality ofshift-invariant feature quantities respectively belong. The classes areobtained by clustering the plurality of shift-invariant featurequantities which are: calculated from a plurality of pixels included inthe normal images each of which does not include a lesion site; andrepresented by the predetermined base vectors. The details of theselecting unit 106 will be described later.

The shift-invariant feature quantity calculating unit 105 includes animage feature quantity calculating unit 110 and a base representing unit120.

The image feature quantity calculating unit 110 calculates, for eachpixel, an examination image feature quantity vector which has, aselements, a plurality of shift-invariant feature quantities, from theexamination image obtained by the examination image obtaining unit 100.

The base representing unit 120 obtains normal image base vectors fromthe base vector storage 130, and transforms the examination imagefeature quantity vector calculated by the image feature quantitycalculating unit 110 into base representation. More specifically, thebase representing unit 120 calculates, for each pixel of the examinationimage, coefficients used to represent the examination image featurequantity vector in the linear combination of the normal image basevectors, and calculates an examination image base coefficient vectorwhich has the calculated coefficients as elements.

The selecting unit 106 includes: a nearest neighbor vector obtainingunit 140; a relative positional relationship obtaining unit 160; and apixel selecting unit 107.

The nearest neighbor vector obtaining unit 140 compares (i) theexamination image base coefficient vector calculated by the baserepresenting unit 120 and (ii) each of the local feature normalstructure vectors which are stored in the local feature normal structuredatabase 150 and calculated from a plurality of normal images inadvance, and obtains, for each of the pixels of the examination image,the local feature normal structure vector of which the distance from theexamination image base coefficient vector is shortest. At this time, thenearest neighbor vector obtaining unit 140 obtains (i) the class numbersof classes to which the obtained local feature normal structure vectorsrespectively belong and (ii) a distance between the examination imagebase coefficient vector and each of the obtained local feature normalstructure vectors.

The relative positional relationship obtaining unit 160 obtains thematching degree of the relative positional relationship between thelocal feature quantities of the examination image, from the relativepositional relationship normal structure database 170 in which thestructural matching degree between the local feature quantitiescalculated in advance from the normal image is stored. In sum, therelative positional relationship obtaining unit 160 obtains thestructural matching degree of each of the pixels of the examinationimage, using the class number obtained by the nearest neighbor vectorobtaining unit 140, from the relative positional relationship normalstructure database 170.

The pixel selecting unit 107 is a processing unit which selects a pixelon the examination image, of which the distance obtained by the nearestneighbor vector obtaining unit 140 is greater than or equal to the firstthreshold, or of which the structural matching degree obtained by therelative positional relationship obtaining unit 160 is smaller than orequal to the second threshold, and includes a lesion determining unit180.

The lesion determining unit 180 obtains a threshold for determining alesion from the lesion determination threshold database 190 to performlesion determination through threshold processing. As the threshold, twotypes of thresholds are used, that is, the first threshold forperforming abnormality determination using the local feature quantityand the second threshold for performing abnormality determination usingthe structural relationship. The lesion determining unit 180 determines,as the pixel of the lesion site, the pixel of the examination imagecorresponding to the examination image base coefficient vector, when thedistance between (i) the examination image base coefficient vectorobtained by the nearest neighbor vector obtaining unit 140 and (ii) thelocal feature normal structure vector that is most similar to theexamination image base coefficient vector is greater than or equal tothe first threshold for determining the local feature quantity. Inaddition, the lesion determining unit 180, when the structural matchingdegree obtained by the relative positional relationship obtaining unit160 is smaller than or equal to the second threshold for determining thestructural relationship, determines the pixel of the examination imagecorresponding to the structural matching degree as the pixel of thelesion site.

The output unit 200 outputs the result of the determination performed bythe lesion determining unit 180. For example, the output unit 200displays the portion determined as the lesion, on a display screen suchas a display apparatus.

FIG. 3 is a block diagram which illustrates a configuration of an imageprocessing apparatus which generates data to be used by the imageprocessing apparatus 310 illustrated in FIG. 2. More specifically, animage processing apparatus 400 generates a base vector stored in thebase vector storage 130, the local feature normal structure database150, and the relative positional relationship normal structure database170.

The image processing apparatus 400 includes: a normal image obtainingunit 210; the image feature quantity calculating unit 110; a principalcomponent analysis unit 220; a base vector output unit 230; the basevector storage 130; a clustering unit 240; a center vector output unit250; the local feature normal structure database 150; a relativepositional relationship calculating unit 260; a relative positionalrelationship output unit 270; and the relative positional relationshipnormal structure database 170.

The normal image obtaining unit 210 reads, as a normal image, an imageconfirmed in advance that there is no abnormality by a doctor. In otherwords, the normal image obtaining unit 210 obtains a plurality of normalimages which are images including no lesion.

The image feature quantity calculating unit 110 calculates an imagefeature quantity from the normal image obtained by the normal imageobtaining unit 210, and generates a normal image feature quantityvector. In other words, the image feature quantity calculating unit 110calculates, for each pixel, a normal image feature quantity vector whichhas, as elements of a vector, a plurality of shift-invariant featurequantities, from each of the normal images obtained by the normal imageobtaining unit 210.

The principal component analysis unit 220 performs principal componentanalysis on the normal image feature quantity vector obtained by theimage feature quantity calculating unit 110, and obtains normal imagebase vectors and a normal image base coefficient vector which isobtained by representing the normal image feature quantity vector bybase representation. More specifically, the principal component analysisunit 220 performs principal component analysis on the normal imagefeature quantity vectors calculated, by the image feature quantitycalculating unit 110, from a plurality of pixels included in a pluralityof normal images, thereby obtaining (i) normal image base vectors whichare the base vectors of the normal image feature quantity vectors and(ii) normal image base coefficient vectors resulting from baseconversion performed on the normal image feature quantity vectors usingthe normal image base vectors.

The base vector output unit 230 stores the normal image base vectorsobtained by the principal component analysis unit 220 into the basevector storage 130.

The clustering unit 240 performs clustering on the normal image basecoefficient vectors obtained by the principal component analysis unit220, calculates, for each class, a center vector of at least one of thenormal image base coefficient vectors included in the class, and assignsa class number to at least one of the normal image base coefficientvectors included in the class.

The center vector output unit 250 writes the center vector of each ofthe classes obtained by the clustering unit 240, as a local featurenormal structure vector representing a local normal structure, togetherwith the class number into the local feature normal structure database150.

The relative positional relationship calculating unit 260 calculates, asa structural matching degree, the relative positional relationship ofeach of the classes, based on the class number of each of the normalimage base coefficient vectors obtained by the clustering unit 240.

The relative positional relationship output unit 270 writes the relativepositional relationship of the classes obtained by the relativepositional relationship calculating unit 260, as the structural matchingdegree, onto the relative positional relationship normal structuredatabase 170.

The following describes in detail the operations performed by the imageprocessing apparatus 310 and the image processing apparatus 400according to the exemplary embodiment.

[Preparation of Database]

In the exemplary embodiment, a database of a normal structure based on ashift-invariant local image feature quantity is constructed so as toeliminate the need for position adjustment. More specifically, the imageprocessing apparatus 400 stores the shift-invariant local image featurequantity obtained from the normal image and the relative positionalrelationship between the local feature quantities, into the localfeature normal structure database 150 and the relative positionalrelationship normal structure database 170, respectively. The imageprocessing apparatus 310 eliminates the need for position adjustment ofthe overall image, by employing the shift-invariant local featurequantity as a criteria for lesion determination. In addition, the imageprocessing apparatus 310 enables determination of being normal orabnormal in the structure of human body, which cannot be determined byonly the local feature quantity, by employing the relative positionalrelationship between local feature quantities as the criteria for lesiondetermination.

The following describes the procedure of generation of the normalstructure database performed by the image processing apparatus 400, withuse of the flowchart illustrated in FIG. 4. Although a plain chest X-rayimage is used as a subject medical image in the exemplary embodiment,equivalent processing is possible by using a medical image such as acomputer tomography (CT) image, a magnetic resonance imaging (MRI)image, a positron emission tomography (PET) image, a pathological image.

In Step S10, the normal image obtaining unit 210 obtains one normalimage for generating a normal structure database. As the normal image, amedical image on which a doctor has made a diagnosis and which has beenconfirmed that there is no abnormality. Medical images captured in alarge number of hospitals are currently stored in picture archiving andcommunication systems (PSCS) together with diagnosis results. For thatreason, it is easy to collect normal images in large numbers. It isdesirable to use more than hundreds and thousands of normal images togenerate a normal structure database.

In Step S11, the image feature quantity calculating unit 110 calculates,from the normal image obtained in Step S10, a shift-invariant featurequantity that is a shift-invariant image feature quantity, vectorizesimage information using the shift-invariant feature quantity, andoutputs a normal image feature quantity vector f. In the exemplaryembodiment, a wavelet coefficient is used as the shift-invariant imagefeature quantity.

FIG. 5 illustrates an example of calculating a shift-invariant imagefeature quantity according to wavelet transformation. The image featurequantity calculating unit 110 performs multiresolution representation ofscale t on a normal image using the wavelet transformation.

In scale 1, a luminance difference between adjacent pixels iscalculated, and smoothing is performed between a plurality of pixelswhen shifting to scale 2.

Although a luminance difference between adjacent pixels is calculated inscale 2 as well, each of the pixels in scale 2 is a result of smoothingthe pixels in scale 1, and thus the frequency component is lower inscale 2. Thus, by proceeding with the calculation from scale 1 to scalet (t is an integer greater than or equal to two), the waveletcoefficients V, H, and D are calculated in each scale while graduallyshifting from the high-frequency component to the low frequencycomponent. The image feature quantity calculating unit 110 generates,for each of the pixels, a spatial frequency vector F which includes (i)the wavelet coefficients V, H, and D calculated in each stage and (ii) aluminance average value L calculated from an image of scale t. In otherwords, the dimension number of the spatial frequency vector F is (3t+1)dimension.

When Haar's mother wavelet is employed, as illustrated in FIG. 6A, Vdenotes the luminance difference value of the pixel 60 to be processedfrom the right adjacent pixel 61, H denotes the luminance differencevalue of the pixel 60 to be processed from the lower adjacent pixel 62,D denotes the luminance difference value from the diagonally lower rightadjacent pixel 63, and L denotes the luminance average value of fourpixels including the pixel 60 to be processed, the right adjacent pixel61, the lower adjacent pixel 62, and the diagonally lower right adjacentpixel 63. FIG. 6A corresponds to scale 1 and FIG. 6B corresponds toscale 2. The examination image of scale 2 is an image in which eachpixel has the luminance average value of four pixels of the examinationimage of scale 1. In other words, in scale 2, an output L which is theluminance average value of four pixels of scale 1 is the luminance valueof a block which is the target for calculating the luminance differencevalue. The output V of scale 2 is the luminance difference value betweenthe block 64 and the right adjacent block 65, the output H of scale 2 isthe luminance difference value between the block 64 and the loweradjacent block 66, and the output D of scale 2 is the luminancedifference value between the block 64 and the diagonally lower rightadjacent block 67. In addition, the output L of scale 2 is the luminanceaverage value of four blocks, from the block 64 to the diagonally lowerright adjacent block 67.

Through the processes described above, when the wavelet transformationis employed, the normal image feature quantity vector f is calculated asthe spatial frequency vector F illustrated in FIG. 5.

It is to be noted that, although the wavelet coefficient is used as theshift-invariant image feature quantity in the exemplary embodiment, theshift-invariant feature quantity is not limited to this, and anarbitrary shift-invariant feature quantity may be employed. For example,it is possible to use SIFT feature quantity, HLAC feature quantity, HOGfeature quantity, and so on can be used as the shift-invariant featurequantity.

In Step S12, the normal image obtaining unit 210 determines whether ornot a normal image which has not been obtained is present. When there isa normal image which has not been obtained is present, the process goesbacks to Step S10, and the process of obtaining a not-yet-obtained imageand extracting an image feature (S11) is repeated. When all of thenormal images have already been obtained, the process proceeds to StepS13.

In Step S13, the principal component analysis unit 220 performsprincipal component analysis on the normal image feature quantity vectorobtained in the process in or prior to Step S12, and calculates thenormal image average vector g, the normal image base vectors matrix B,and the normal image base coefficient vector. FIG. 7 illustrates anexample of the normal image feature quantity vector obtained in or priorto Step S12. When a normal image has the width of W and the height of H,(W×H) normal image feature quantity vectors are calculated from a singlenormal image. When the number of normal images is Q, (W×H×Q) normalimage feature quantity vectors are obtained from Q normal images. Thedimension number of the normal image feature quantity is assumed to be ndimension.

The normal image average vector g is obtained by calculating an averagevalue for each element of the normal image feature quantity vectors.

The normal image base vectors matrix B is calculated as eigen vectorsb1, b2, . . . , bn which are the solutions of the simultaneous equationof Expression 2 below, by the analysis of principal component.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack & \; \\{{{S\; b_{1}} = {\lambda_{1}b_{1}}}{{S\; b_{2}} = {\lambda_{2}b_{2}}}\vdots{{S\; b_{n}} = {\lambda_{n}b_{n}}}} & {{Expression}\mspace{14mu} 2}\end{matrix}$

Here, the matrix S is a variance-covariance matrix, and is provided byExpression 3 below.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 3} \right\rbrack & \; \\{S = \begin{pmatrix}s_{1}^{2} & {s_{1}s_{2}} & \ldots & {s_{1}s_{n}} \\{s_{1}s_{2}} & s_{2}^{2} & \ldots & {s_{2}s_{n}} \\\vdots & \vdots & \ddots & \vdots \\{s_{1}s_{n}} & {s_{2}s_{n}} & \ldots & s_{n}^{2}\end{pmatrix}} & {{Expression}\mspace{14mu} 3}\end{matrix}$

In the expression, si is a variance of i-dimension element of the imagefeature quantity vector. As described above, the image feature quantityvector is obtained for the number of (W×H×Q), and thus i-dimensionelement of the image feature quantity vector is present for the numberof (W×H×Q). Thus, si is the variance of them.

In addition, the eigen value A is provided by Expression 4.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 4} \right\rbrack & \; \\{{\begin{matrix}{s_{1}^{2} - \lambda} & {s_{1}s_{2}} & \ldots & {s_{1}s_{n}} \\{s_{1}s_{2}} & {s_{2}^{2} - \lambda} & \ldots & {s_{2}s_{n}} \\\vdots & \vdots & \ddots & \vdots \\{s_{1}s_{n}} & {s_{2}s_{n}} & \ldots & {s_{n}^{2} - \lambda}\end{matrix}} = 0} & {{Expression}\mspace{14mu} 4}\end{matrix}$

The eigen value λ is obtained for the number of n, and denoted as λ1,λ2, . . . , λn in descending order.

The normal image base coefficient vector a is calculated according toExpression 5.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 5} \right\rbrack & \; \\{\alpha = {\left. {B^{- 1}\left( {f - g} \right)}\Leftrightarrow\begin{pmatrix}\alpha_{1} \\\alpha_{2} \\\vdots \\\alpha_{n}\end{pmatrix} \right. = {\left. {\left( {b_{1}\mspace{14mu} b_{2}\mspace{14mu}\ldots\mspace{14mu} b_{n}} \right)^{- 1}\left( {f - g} \right)}\Leftrightarrow\begin{pmatrix}\alpha_{1} \\\alpha_{2} \\\vdots \\\alpha_{n}\end{pmatrix} \right. = {\begin{pmatrix}b_{1,1} & b_{2,1} & \ldots & b_{n,1} \\b_{1,2} & b_{2,2} & \ldots & b_{n,2} \\\vdots & \vdots & \vdots & \vdots \\b_{1,n} & b_{2,n} & \ldots & b_{n,n}\end{pmatrix}^{- 1}\begin{pmatrix}{f_{1} - g_{1}} \\{f_{2} - g_{2}} \\\vdots \\{f_{n} - g_{n}}\end{pmatrix}}}}} & {{Expression}\mspace{14mu} 5}\end{matrix}$

Here, the vector f is the normal image feature quantity vector.

In Step S14, the base vector output unit 230 stores the normal imageaverage vector g and the normal image base vector matrix B which havebeen obtained in Step S13, into the base vector storage 130.

In Step S15, the clustering unit 240 performs clustering on the normalimage base coefficient vectors obtained in Step S13, and determines theclass number of each class. Furthermore, the center vector of each classis obtained concurrently.

In the exemplary embodiment, k-means method is employed as a techniqueof clustering. According to K-means method, all of the normal image basecoefficient vectors are classified into classes by repeating calculationof the center vector of each of the classes and classifying the normalimage base coefficient vectors into classes (allocation into a classhaving the shortest distance from the center vector). FIG. 8 illustratesan example of determining the class number of the normal image basecoefficient vector. For example, it is shown that the normal image basecoefficient vector of a pixel of which a coordinate (x, y) is (1, 1) inthe first image is classified into the class having the class number of0.

In Step S16, the center vector output unit 250 stores the center vectorof each of the classes calculated in Step S15, as the local normalstructure, into the local feature normal structure database 150.

FIG. 9 illustrates an example of a local feature normal structuredatabase. As illustrated in FIG. 9, the local feature normal structuredatabase 150 holds the class number and the center vector thereof as aset.

In Step S17, the relative positional relationship calculating unit 260calculates a relative positional relationship between the local featurequantities, as the structural relationship (structural matching degree),based on the class number of the normal image feature quantity vectorobtained in Step S15.

FIG. 10 shows an example in which the structural relationship isrequired for properly determining a lesion site. In FIG. 10, the region(a) is a lesion site, and the region (b) is a normal portion, and whenthey are compared by the local image feature quantities, they havesubstantially the same pattern. For that reason, similar image featurequantities are extracted from each of the regions. Thus, it is difficultto distinguish the region (a) from the region (b) only by the localimage feature quantities.

In the exemplary embodiment, the structural relationship is employed inorder to solve the above-described problem. As illustrated in FIG. 11,it is possible to distinguish between normality and abnormality of animage by taking into account the relative positional relationshipbetween the local feature quantities extracted from the image. In thecase of FIG. 10, by taking the structural relationship with thesurrounding area into account, it can be seen that the region (a) is anabnormal portion in which a tumor is developed, and the region (b) is apart of the normal portion.

In the exemplary embodiment, by capturing the relative positionalrelationship with the surrounding area as a structure, it is possible todetermine whether the region (a) and the region (b) are present in theright or left of the spine, for example.

According to the exemplary embodiment, the probability that classes areadjacent to each other is used as the relative positional relationship(structural matching degree) between the local feature quantities. InStep S15, the class number of the normal image feature quantity vectorof each coordinate is calculated. For that reason, the relativepositional relationship calculating unit 260 is capable of obtaining theprobability that the classes are adjacent to each other in a normalimage.

FIG. 12 is a table which indicates, when focusing on a pixel, theprobability that each class appears in a right to the right of the focuspixel. For example, when a focus pixel is a pixel of the class 1, theprobability that the class 1 appears in the right pixel is 0.8. Inaddition, the probability that the class 2 appears in the right pixel is0.09. When a local feature quantity appears in the examination image, itis possible to determine the structural normality as viewed includingthe surrounding area, by employing the table as described above. It isindicated that a focus pixel with a large value (probability) in thetable in FIG. 12 has the relative positional relationship between thelocal feature quantities frequently appearing in the normal image. Forthat reason, it is highly likely that the focus pixel has a normalstructure. On the other hand, it is indicated that a focus pixel with asmall value in the table is highly likely to be structurally abnormal.According to the exemplary embodiment, the table that represents such astructural relationship is obtained for the surrounding eightdirections.

It is to be noted that the table that represents the structuralrelationship is not necessarily limited to the table obtained for thesurrounding eight directions, and it is also possible to obtain thetable using the relationship with a pixel located away by two or morepixels, or using the relationship with a class from which there is adistance not to overlap the region covered by the wavelettransformation.

In addition, it is also possible to regard adjacent similar classes asthe same group to make a database of the relationship with an adjacentdifferent class. FIG. 13 shows the probability that a class differentfrom the class of the focus pixel appears, for each class number, whenthe pixels are traced rightward from the focus pixel. For example, it isindicated that, in the case where the class number of the focus pixel is1, the probability that a pixel of the class 2 appears first when tracedrightward is 0.09, and, the probability that a pixel of the class 3appears first when traced rightward is 0.02. The probability thatadjacent pixels are in the same class is high in some cases. For thatreason, the structure is more easily captured, by holding therelationship with the local image feature quantity obtained from aposition that is distant to some extent, and thus it is possible toincrease accuracy. In other words, the probability that the same classappears in adjacent pixels is high in some cases. However, according tothis configuration, it is possible to perform lesion detection, usingappearance probability of classes of pixels which are present atpositions that are mutually distant to some extent and have differentclass numbers, as a matching degree. For that reason, the structure ofthe examination image is more easily captured, and thus it is possibleto increase the accuracy in lesion detection.

In Step S18, the relative positional relationship output unit 270 storesthe relative positional relationship between the local featurequantities in the normal image calculated in Step S17, into the relativepositional relationship normal structure database 170.

[Lesion Detection]

The following describes the procedure of lesion detection performed bythe image processing apparatus illustrated in FIG. 2, with use of theflowchart illustrated in FIG. 14.

In Step S20, the examination image obtaining unit 100 obtains anexamination image which is a subject for the lesion detection, from theexamination image storage 300.

In Step S21, the image feature quantity calculating unit 110, for theexamination image obtained in Step S20, vectorizes image informationusing the shift-invariant feature quantity, and outputs an examinationimage feature quantity vector fp. Step S21 can be implemented byperforming a similar process to Step S11.

In Step S22, the base representing unit 120 transforms the examinationimage feature quantity vector fp calculated in Step S21 to base vectorrepresentation, thereby obtaining the examination image base coefficientvector op.

First, the base representing unit 120 obtains, from the base vectorstorage 130, the normal image average vector g and the normal image basevector matrix B for transformation into base representation. The baserepresenting unit 120 transforms the examination image feature quantityvector fp to the examination image base coefficient vector op accordingto Expression 6 below.

$\begin{matrix}{\mspace{79mu}\left\lbrack {{Math}.\mspace{14mu} 6} \right\rbrack} & \; \\{\mspace{79mu}{\alpha_{p} = {\left. {B^{- 1}\left( {f_{p} - g} \right)}\mspace{79mu}\Leftrightarrow\begin{pmatrix}\alpha_{p,1} \\\alpha_{p,2} \\\vdots \\\alpha_{p,n}\end{pmatrix} \right. = {\left. {\left( {b_{1}\mspace{14mu} b_{2}\mspace{14mu}\ldots\mspace{14mu} b_{n}} \right)^{- 1}\left( {f_{p} - g} \right)}\Leftrightarrow\begin{pmatrix}\alpha_{p,1} \\\alpha_{p,2} \\\vdots \\\alpha_{p,n}\end{pmatrix} \right. = {\begin{pmatrix}b_{1,1} & b_{2,1} & \ldots & b_{n,1} \\b_{1,2} & b_{2,2} & \ldots & b_{n,2} \\\vdots & \vdots & \vdots & \vdots \\b_{1,n} & b_{2,n} & \ldots & b_{n,n}\end{pmatrix}^{- 1}\begin{pmatrix}{f_{p,1} - g_{1}} \\{f_{p,2} - g_{2}} \\\vdots \\{f_{p,n} - g_{n}}\end{pmatrix}}}}}} & {{Expression}\mspace{14mu} 6}\end{matrix}$

Expression 6 has the same structure as Expression 5, and the examinationimage base coefficient vector op is calculated by providing theexamination image feature quantity vector fp in place of the normalimage feature quantity vector f of Expression 5.

As described above, the base representing unit 120 transforms the imagefeature quantity vector of the examination image to the baserepresentation, thereby enabling comparison between the examinationimage and the normal image.

In Step S23, the nearest neighbor vector obtaining unit 140 obtains,from the local feature normal structure database 150, a local featurenormal structure vector having the shortest distance from theexamination image base coefficient vector obtained in Step S22 and aclass number thereof.

At this time, the nearest neighbor vector obtaining unit 140concurrently calculates the distance between the obtained local featurenormal structure vector and the examination image base coefficientvector. The vector stored in the local feature normal structure databaseis generated from a normal image. For that reason, when the localfeature normal structure vector having a short distance is present ingreater number, the examination image is likely to be normal.

In Step S24, the relative positional relationship obtaining unit 160obtains, from the relative positional relationship normal structuredatabase 170, the matching degree of the relative positionalrelationship between the local feature quantities, based on the classnumber of the local feature quantity obtained in Step S23. In Step S23,the class number of the local feature normal structure vector which ismost similar to the examination image base coefficient vector in each ofthe pixels in the examination image. The relative positionalrelationship between the class numbers is obtained from the relativepositional relationship normal structure database 170, thereby obtainingthe structural matching degree of the focus pixel of the examinationimage. Since the relative positional relationship of each of the classes(the probability that the classes are adjacent to each other) calculatedfrom the normal image is stored in the relative positional relationshipnormal structure database 170, the structural matching degree of thefocus pixel in the examination image to the normal image is higher asthe value in the database is larger. According to the exemplaryembodiment, as the relative relationship, the relationship (structuralmatching degree) between the focus pixel and the pixels in 8 surroundingmasses, respectively from the relative positional relationship normalstructure database. As the final matching degree, the average value orthe maximum value of the structural matching degree of each of the 8surrounding masses is employed.

In Step S25, the lesion determining unit 180 performs lesiondetermination based on the distance from the nearest neighbor vectorobtained in Step S23, and the structural relationship between the localfeature quantities obtained in Step S24. According to the exemplaryembodiment, the lesion determination is performed using twodetermination criteria; that is, the local feature quantity and thestructural matching degree.

For the lesion determination based on the local features, the distancebetween the examination image base coefficient vector and the localfeature normal structure vector obtained in Step S23 is used. When thedistance is greater than or equal to the first threshold, the localfeature quantity of the focus pixel is the feature which does not appearin the normal image, and thus it is possible to determine the focuspixel as the lesion site.

The threshold for determining the presence or absence of a lesion iscalculated from previous cases, and stored in the lesion determinationthreshold database 190.

FIG. 15 is a diagram illustrating an example of obtaining adetermination threshold for determining the presence or absence of alesion. A large number of lesion images each containing a lesion siteconfirmed by a doctor are obtained from previous cases, and the positionof the lesion site is supervised in advance. In other words, theposition of the lesion site is specified in advance. Next, the imagefeature quantity calculating unit 110 calculates, for each pixel, alesion image feature quantity vector which has shift-invariant featurequantities, as elements of a vector. At this time, the image featurequantity calculating unit 110 calculates the lesion image featurequantity vector by dividing the lesion image into a lesion site and anormal portion, and obtains (i) a lesion site image feature quantityvector fd from the lesion site and (ii) a normal portion image featurequantity vector fn from the normal portion.

The lesion site image feature quantity vector fd and the normal portionimage feature quantity vector fn are assigned to the vector fp ofExpression 6 by the base representing unit 120, and transformedrespectively to a lesion site base coefficient vector ad and a normalportion base coefficient vector an.

Next, the nearest neighbor vector obtaining unit 140 searches for anormal image base coefficient vector a which is closest to each of thelesion site base coefficient vectors ad and the normal portion basecoefficient vectors an, and obtains the respective distances.

Through the processes described above, the distance in the normalportion and the distance in the abnormal portion are respectivelycalculated. The first threshold is determined which most effectivelyseparates the lesion site from a normal portion when lesiondetermination is performed based on the threshold, by using theinformation.

In the evaluation of the structural matching degree, the lesiondetermining unit 180 performs lesion determination based on thestructural matching degree obtained in Step S24. As with the case of thelocal feature quantity, the second threshold which is the lesiondetermination threshold is obtained in advance, and the lesiondetermination is carried out by threshold processing. By determining thestructural matching degree in addition to the lesion determinationaccording to the local feature quantity, it is possible to determine thestructural matching degree which cannot be determined only with thelocal feature quantity, such as that illustrated in FIG. 10.

In Step S26, the output unit 200 outputs the result of determination ofthe lesion site obtained in Step S25. For example, the output unit 200,when it is determined that there is a lesion site, replaces a pixelvalue of the lesion site with a pixel value of a specific color, anddisplays the presence of the lesion site and the position thereof as animage.

Through the processes as described above, lesion detection is madepossible without requiring position adjustment between images, byemploying the shift-invariant local image feature quantity. Furthermore,lesion detection is made possible which can determine as structurallybeing normal or abnormal which cannot be determined only by the localfeature quantity, by employing the relative positional relationshipbetween the local feature quantities as a structure.

According to the exemplary embodiment, the shift-invariant featurequantity calculated from a normal image and the shift-invariant featurequantity calculated from an examination image are represented by thesame base vectors. This eliminates the necessity of setting a landmarkfor position adjustment between the normal image and the examinationimage. In addition, a pixel is selected using the matching degreebetween (i) the relative positional relationship of classes obtainedfrom the result of clustering the shift-invariant feature quantity of anexamination image and (ii) the relative positional relationship ofclasses obtained from the result of clustering the shift-invariantfeature quantity of a normal image. Thus, by employing a relativepositional relation between local image feature quantities, asknowledge, it is possible to detect a lesion more accurately, takinginto consideration the structure of human body that cannot berepresented only by a local feature.

More specifically, the examination image and the normal image arecompared between the base coefficient vectors, using the shift-invariantfeature quantity. For that reason, it is possible to determine thepresence or absence of a lesion without performing the positionadjustment between the normal image and the examination image. Since theposition adjustment processing is not required, there is no decrease inthe determination accuracy of a lesion site caused by the difference ina setting method of a landmark or variation in a setting position, andthus it is possible to provide an image processing apparatus with highaccuracy in determining a lesion site. Furthermore, a pixel, on anexamination image, of which the structural matching degree indicatingthe relative positional relationship of classes in the normal image islower than or equal to the second threshold is selected. For thatreason, it is possible to select a pixel, on the examination image,which has a structure different from a structure of the normal image.Thus, it is possible to detect a lesion more accurately, taking intoconsideration the structure of human body which cannot be representedonly by a local feature.

In addition, by employing, as a matching degree, the appearanceprobability of classes of pixels located to be in a predeterminedpositional relationship in the normal image, detection as a lesion siteis made possible when a combination of classes which does not oftenappear in the normal image appears on the examination image.

It is to be noted that, each of the structural elements in theabove-described exemplary embodiment may be configured in the form of anexclusive hardware product, or may be realized by executing a softwareprogram suitable for the structural element. Each of the structuralelements may be realized by means of a program executing unit, such as aCPU and a processor, reading and executing the software program recordedon a recording medium such as a hard disk or a semiconductor memory.Here, software that accomplishes the image coding apparatus according toeach of the above-described embodiments is a program as below.

More specifically, the program causes a computer to execute obtaining anexamination image which is an image of an examination subject;calculating, for each pixel, a shift-invariant feature quantity which isrepresented by predetermined base vectors, from the examination imageobtained; selecting, on the examination image, a pixel having a matchingdegree lower than or equal to a predetermined threshold, between (i) arelative positional relationship of classes in a plurality of normalimages each of which does not include a lesion site and (ii) a relativepositional relationship of the classes to which a plurality ofshift-invariant feature quantities respectively belong in theexamination image, the classes being obtained by clustering theplurality of shift-invariant feature quantities which are: calculatedfrom a plurality of pixels included in the normal images; andrepresented by the predetermined base vectors; and outputting a resultof the selection.

Although only some exemplary embodiments have been described in detailabove, those skilled in the art will readily appreciate that variousmodifications may be made in these exemplary embodiments withoutmaterially departing from the principles and spirit of the inventiveconcept, the scope of which is defined in the appended Claims and theirequivalents.

One or more exemplary embodiments of the present disclosure areapplicable in an image processing apparatus and the like for identifyinga lesion site from a medical image.

The invention claimed is:
 1. An image processing apparatus comprising: anon-transitory memory storing a program; and a hardware processor thatexecutes the program, the program causing the image processing apparatusto operate as: an examination image obtaining unit configured to obtainan examination image which is an image of an examination subject; ashift-invariant feature quantity calculating unit configured tocalculate, for each pixel, a shift-invariant feature quantity which isrepresented by predetermined base vectors, from the examination imageobtained by the examination image obtaining unit; a selecting unitconfigured to select, on the examination image, a pixel of which (i) adistance between an examination image base coefficient vector and alocal feature normal structure vector is greater than or equal to afirst threshold or (ii) a matching degree is lower than or equal to apredetermined threshold, between (i) a relative positional relationshipof classes in a plurality of normal images each of which does notinclude a lesion site and (ii) a relative positional relationship of theclasses to which a plurality of shift-invariant feature quantitiesrespectively belong in the examination image, the classes being obtainedby clustering the plurality of shift-invariant feature quantities whichare: calculated from a plurality of pixels included in the normalimages; and represented by base vectors identical to the predeterminedbase vectors; and an output unit configured to output a result of theselection performed by the selecting unit.
 2. The image processingapparatus according to claim 1, wherein the shift-invariant featurequantity calculating unit includes: an image feature quantitycalculating unit configured to calculate, for each pixel, an examinationimage feature quantity vector which has a plurality of theshift-invariant feature quantities as elements of a vector, from theexamination image obtained by the examination image obtaining unit; anda base representing unit configured to calculate, for each pixel of theexamination image, (i) coefficients used to represent the examinationimage feature quantity vector in a linear combination of normal imagebase vectors which are: base vectors of a plurality of normal imagefeature quantity vectors each of which is calculated from the pixelsincluded in the normal images; and base vectors of the normal imagefeature quantity vectors each having the shift-invariant featurequantities as the elements of a vector, and (ii) an examination imagebase coefficient vector having the calculated coefficients as theelements of the vector, the selecting unit includes: a nearest neighborvector obtaining unit configured to (i) obtain, for the each pixel ofthe examination image, a local feature normal structure vector which ismost similar to an examination image base coefficient vector from alocal feature normal structure database, and (ii) obtain: a class numberof a class to which the obtained local feature normal structure vectorbelongs; and a distance between the examination image base coefficientvector and the obtained local feature normal structure vector, the localfeature normal structure database storing, for each of classes obtainedby clustering a plurality of normal image base coefficient vectors, thelocal feature normal structure vector which is a center vectorrepresenting at least one normal image base coefficient vector thatbelongs to the class, together with a class number of the class, theplurality of the normal image base coefficient vectors being obtainedfrom the normal image feature quantity vectors and each having, aselements, coefficients used to represent the normal image featurequantity vector in the linear combination of the normal image basevectors; a relative positional relationship obtaining unit configured toobtain a structural matching degree of the each pixel of the examinationimage, using the class number obtained by the nearest neighbor vectorobtaining unit, from a relative positional relationship normal structuredatabase which stores, as the structural matching degree, a relativepositional relationship of the classes in the normal images, therelative positional relationship being obtained from class numbersresulting from clustering performed on the normal image feature quantityvectors; and a pixel selecting unit configured to select, on theexamination image, a pixel of which (i) the distance obtained by thenearest neighbor vector obtaining unit is greater than or equal to afirst threshold or (ii) the structural matching degree obtained by therelative positional relationship obtaining unit is smaller than or equalto a second threshold, and the output unit is configured to output aresult of the selection performed by the selecting unit.
 3. The imageprocessing apparatus according to claim 2, wherein the pixel selectingunit includes a lesion determining unit configured to determine, on theexamination image, a pixel of which (i) the distance obtained by thenearest neighbor vector obtaining unit is greater than or equal to afirst threshold or (ii) the structural matching degree obtained by therelative positional relationship obtaining unit is smaller than or equalto a second threshold, as a pixel of a lesion site, and the output unitis configured to output a result of the determination performed by thelesion determining unit.
 4. The image processing apparatus according toclaim 2, wherein the relative positional relationship normal structuredatabase stores an appearance probability of classes of pixels locatedto be in a predetermined positional relationship, as the structuralmatching degree, and the relative positional relationship obtaining unitis configured to (i) identify classes of the pixels located to be in thepredetermined positional relationship in the examination image, usingthe class number obtained by the nearest neighbor vector obtaining unit,and (ii) obtain the appearance probability of the classes of the pixelslocated to be in the predetermined positional relationship from therelative positional relationship normal structure database, to obtainthe structural matching degree of each of the pixels included in theexamination image.
 5. The image processing apparatus according to claim2, wherein the relative positional relationship normal structuredatabase stores, as a structural matching degree, an appearanceprobability of classes of pixels aligned in a predetermined directionand having different class numbers, and the relative positionalrelationship obtaining unit is configured to obtain, from the relativepositional relationship normal structure database, the appearanceprobability of the classes of pixels aligned in the predetermineddirection and having the different class numbers in the examinationimage, using the class number obtained by the nearest neighbor vectorobtaining unit, to obtain a structural matching degree of each of thepixels included in the examination image.
 6. An image processingapparatus comprising: a non-transitory memory storing a program; and ahardware processor that executes the program, the program causing theimage processing apparatus to operate as: a normal image obtaining unitconfigured to obtain a plurality of normal images which are imagesincluding no lesion; an image feature quantity calculating unitconfigured to calculate, for each pixel, a normal image feature quantityvector which has a plurality of shift-invariant feature quantities aselements of a vector, from each of the normal images obtained by thenormal image obtaining unit; a principal component analysis unitconfigured to perform principal component analysis on the normal imagefeature quantity vectors calculated, by the image feature quantitycalculating unit, from pixels included in the normal images to obtain(i) normal image base vectors which are base vectors of the normal imagefeature quantity vectors and (ii) a plurality of normal image basecoefficient vectors resulting from base conversion performed on thenormal image feature quantity vectors using the normal image basevectors; a base vector output unit configured to write, on a base vectorstorage, the normal image base vector obtained by the principalcomponent analysis unit; a clustering unit configured to performclustering on the normal image base coefficient vectors to obtain, foreach class, a center vector which represents at least one of the normalimage base coefficient vectors which belong to the class; a centervector output unit configured to write, on a local feature normalstructure database, the center vector obtained by the clustering unit,together with a class number of a class represented by the centervector; a relative positional relationship calculating unit configuredto calculate, as a structural matching degree, a relative positionalrelationship of classes in the normal images, the relative positionalrelationship being obtained from the class number; a relative positionalrelationship output unit configured to write, as the structural matchingdegree, a relative positional relationship calculated by the relativepositional relationship calculating unit, between classes in the normalimages, on a relative positional relationship normal structure database;and a selecting unit configured to select, on the examination image, apixel of which (i) a distance between an examination image basecoefficient vector and a local feature normal structure vector isgreater than or equal to a first threshold or (ii) the structuralmatching degree obtained by the relative positional relationshipcalculating unit is smaller than or equal to a second threshold.
 7. Theimage processing apparatus according to claim 1, wherein theshift-invariant feature quantity includes a wavelet coefficient, ahigher order local autocorrelation (HLAC) feature quantity, ascale-invariant feature transform (SIFT) feature quantity, or ahistogram of oriented gradients (HOG) feature quantity.
 8. The imageprocessing apparatus according to claim 1, wherein the image is one of aradiological image, an ultrasound image, and a pathological specimenimage.
 9. An image processing method comprising: obtaining anexamination image which is an image of an examination subject;calculating, for each pixel, a shift-invariant feature quantity which isrepresented by predetermined base vectors, from the examination imageobtained; selecting, on the examination image, a pixel of which (i) adistance between an examination image base coefficient vector and alocal feature normal structure vector is greater than or equal to afirst threshold or (ii) a matching degree is lower than or equal to apredetermined threshold, between (i) a relative positional relationshipof classes in a plurality of normal images each of which does notinclude a lesion site and (ii) a relative positional relationship of theclasses to which a plurality of shift-invariant feature quantitiesrespectively belong in the examination image, the classes being obtainedby clustering the plurality of shift-invariant feature quantities whichare: calculated from a plurality of pixels included in the normalimages; and represented by base vectors identical to the predeterminedbase vectors; and outputting a result of the selection.
 10. Anon-transitory computer-readable recording medium having a computerprogram recorded thereon for causing the computer to execute the imageprocessing method according to claim 9.